Editorial: Evaluating knowledge engineering techniques

We argue below that knowledge engineers (KEs) use a range of techniques ( ). Yet more techniques are constantly being evolved. How sure are we that any of the current KE techniques either work at all or work better than past practice? Currently, our basic question has no clear answer. It is not easy to evaluate software development techniques, be they from software engineering or knowledge engineering (see the list of problems in [Shadbolt, O’Hara & Crow 2000]). However, many successful evaluations have been performed, as evidenced by numerous KE case studies studies (Figure 1) and the wealth of material on KE evaluation (Figure 2). In our experience, the core problem with performing evaluations is a concern that the whole principle of evaluation is misguided. KE researchers are often reluctant to design and perform rigorous evaluations, asking “what is the general point of such studies”? That is, how can results from one study be relevant to anything else Detailed surveys of evaluation techniques: [Fenton 1991, Cohen 1995].